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Abstract 

Quantifying the complexity of systems consisting of many interacting parts has been an impor- 
tant challenge in the field of complex systems in both abstract and applied contexts. One approach, 
the complexity profile, is a measure of the information to describe a system as a function of the 
scale at which it is observed. We present a new formulation of the complexity profile, which ex- 
pands its possible application to high-dimensional real- world and mathematically defined systems. 
The new method is constructed from the pairwise dependencies between components of the system. 
The pairwise approach may serve as both a formulation in its own right and a computationally 
feasible approximation to the original complexity profile. We compare it to the original complexity 
profile by giving cases where they are equivalent, proving properties common to both methods, 
and demonstrating where they differ. Both formulations satisfy linear superposition for unrelated 
systems and conservation of total degrees of freedom (sum rule). The new pairwise formulation is 
also a monotonically non-increasing function of scale. Furthermore, we show that the new formu- 
lation defines a class of related complexity profile functions for a given system, demonstrating the 
generality of the formalism. 
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There have been various approaches to quantify the complexity of systems in a way 
that is both rigorous and intuitive. The origins of these efforts are rooted in Shannon's 
entropy [1] and Kolmogorov's algorithmic complexity [2]. These two measures have been 
modified and applied in diverse contexts [3-5]. However, a disadvantage of these measures 
is that, being decreasing functions of order, they characterize high entropy systems — usually 
systems at equilibrium — as highly complex systems. On the other hand, reversing this result 
by defining complexity as an increasing function of order, as suggested in other work [6], 
identifies ordered systems as highly complex. Neither equilibrium nor ordered systems match 
our intuitive notion of high complexity [7-10]. 

The literature responding to this dilemma has typically redefined complexity as a convex 
function of order, so that the extremes of order and randomness are both low complexity 
[9-14]. One approach has not sought a single number to quantify complexity and instead 
argues that complexity is inherently a function of the scale of observation of the system 
[15, 16]. Because the complexity of systems lies in the emergence of behaviors from fine 
scale to large scale, this body of work argues that the meaningful measure of complexity 
represents the degree of independence among the subdivisions of the system (the disorder) 
as a function of the size of the subdivisions whose interdependence is considered (scale of 
aggregation of the observation). Because the presence of actions at one scale depends on 
order (coordination) on smaller scales, this measure represents the needed balance between 
order and disorder in a specific and meaningful way. 

The dependency of complexity on scale has been termed the "complexity profile." It has 
been applied to a variety of real world situations [16-19]. An expression for the profile in 
terms of system state probabilities has been provided and justified [15], and it has been 
evaluated for a number of specific models [15, 20-22]. Progress is also being made in embed- 
ding the complexity profile in a theory of structure [23]. For the general case, however, this 
expression has the computational disadvantage of requiring factorial time to compute. In 
this work, we present an alternative formulation of the complexity profile, which (a) requires 
only polynomial time to compute, (b) satisfies key properties of the previous formulation, 
(c) gives the same result as the original formulation for many cases, (d) provides an approx- 
imation of the original formulation where it differs, (e) is always positive and non-increasing 
with increasing scale, which matches intuitive (though not necessarily valid) notions of com- 
plexity, and (f) captures some, but not all, of the high-order dependencies captured by the 
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original formulation. 

Our "pairwise complexity profile" is computed from only the mutual information values 
of each pair of components, while the original formulation requires the mutual information 
of all possible subsets of components. Thus, we capture those dependencies in the system 
that are manifest in the pairwise couplings. Such dependencies are representative of weak 
(and of null) emergence, but not strong emergence, while the original complexity profile 
also captures strong emergent dependencies [24]. The use of pairwise couplings is concep- 
tually similar to the use of a network model or of pairwise interaction models in statistical 
physics. However, systems modeled by such methods may still exhibit higher-order mutual 
information not necessarily derivable from the pairwise couplings. The utility of this new 
formulation depends on the specifics of the problem, as we explore in examples. 

Our formulation of the pairwise complexity profile defines a class of different measures 
with common properties. Distinct complexity profiles result from different functions used to 
measure the coupling between variables — mutual information being our primary example. 
In important ways the pairwise complexity profile exhibits the same behavior across a broad 
set of coupling functions, manifesting the general power of the framework. 

The paper is organized as follows. In Section I, we construct the formula for the pairwise 
complexity profile. In Section II, we show that it provides the same result as the original 
complexity profile for the limiting cases of a totally atomized system, and a totally coher- 
ent system. In Section III we prove that the pairwise formulation in general satisfies two 
theoretical properties of the original complexity profile — superposition of independent sub- 
systems, and conservation of total degrees of freedom — as well as one property not shared 
by the original complexity profile — monotonicity. In Section IV, we discuss the robustness 
of those properties to changing the definition of coupling strength. In Section V, we discuss 
phenomena not represented in the pairwise formulation, which do appear in the original com- 
plexity profile. In Section VI, we bound the computational cost of calculating the pairwise 
complexity profile. 

I. DERIVATION 

We consider a system comprising a set of m components, each of which may assume a 
number of states. The state of one component depends to some extent on the states of the 
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other components. In particular, the likelihood of observing a system state is not equal to 
the product of the probabilities of the relevant component states. Different system structures 
are therefore reflected in distinct probability distributions of the states of the whole system. 
In some contexts, for example, one may infer the probability distribution from a collection of 
systems as a proxy for the ensemble of possible states of a single structure, e.g. a population 
of biological cells. In other contexts, one may employ the ergodic hypothesis to infer the 
probability distribution from states of a system sampled over time, e.g. financial market 
data. 

We can calculate the mutal information between any two components, to obtain a measure 
of their "coupling", a value between and 1. All of the couplings make up a symmetric 
m x m matrix, A. For simplicity, in this paper we assume that each variable has the 
same probability distribution of states when considered independently from the others. In 
the examples we provide below, they are random binary variables, equally likely to be 
or 1. These assumptions were also used in the mathematical development of the original 
complexity profile [15]. 

We construct the pairwise complexity profile from a "variable's eye view" of the system, 
in order to explain its structure. First, we establish what scale and complexity mean for 
an individual variable or component of the system. Intuitively, for a given variable, the 
scale of its action is the number of variables it coordinates with to perform the action. The 
complexity, or the amount of information shared by that group of variables, is given by the 
mutual information between those variables, which is by definition their coupling. Thus, the 
size of the coordinated group depends on the degree of coordination required. Motivated 
by this intuition, we define the "scale of action", ki(p), of the ith variable as a function of 
the coupling threshold p — the amount of coordination necessary for that action. Explicitly, 
ki(p) is the number of variables coupled with the ith variable with coupling strength p or 
greater: 

h(p) = \{j e[l,m]\A v >p}\ (1) 

where <S indicates the number of elements in set S. For p a real number between and 
1, hi is an integer between 1 and m. ki cannot be less than 1 because even for p = 1, the 
highest threshold of coupling, each variable correlates with at least one variable — itself. For 
p > 1, ki(p) =0. Note also that ki(p) is a non-increasing function of p, because if a pair of 
variables has a coupling of at least p\ then it certainly also has a coupling of at least p 2 <P\- 
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The scale of action, ki(p), is the functional inverse of our notion of the complexity profile — 
we are seeking complexity as a function of scale, and ki(p) is a measure of scale as a function 
of complexity. Since the coordination p is a measure of the shared information, it is a 
measure of the complexity of the task. Below, we describe the details of taking the inverse, 
to account for discontinuities, which follows an intuitive mirroring over the k = p diagonal. 
Thus we let 

C i (k)=p(k i ) (2) 

which is non-increasing in all cases, and zero for k > m. This gives us a profile function for 
each variable. Our first, non-normalized ansatz for the complexity profile for the system is 
to sum the functions for all of the variables: 

m 

C(k)=J2 d *( k ) ( 3 ) 

i=l 

However, this must be corrected for multiple counting due to adding together the variables 
that share information. To illustrate the multiple counting, consider a system in which there 
are clusters of highly coupled variables, Eq. (3) counts the information of each cluster once 
for each variable in it, so larger (scale) clusters will also appear to have higher complexity. 
To normalize appropriately, each change (decrease) in the curve defined in Eq. (3) should be 
divided by the scale at which that change is found. Thus, we define the pairwise complexity 
profile as: 

C ^ = E * p(*0 - C{k> + 1)] (4) 

k'=k 

where k is a positive integer. C(k = 1) is the Shannon entropy of the system, and for k > m 
we know that C(k) = C(k) = 0. 

For completeness, we now describe the inversion process from ki(p) to p(ki) = Ci(k) (see 
Figure 1). Our definition for each ki{p) in Eq. (1) is a mapping from the set of non-negative 
real numbers to a set of positive integers. We would like to represent the natural inverse 
function, which should be a mapping from the set of positive integers into the set of real 
numbers. Two issues must be resolved: more than one p may map to the same ki, and there 
are values of p not mapped from any integer ki. We redefine the function ki(p) as a relation 
described as a set of ordered pairs {(p, k)}i. First, we fill in gaps in the k axis direction with 
straight lines between the adjacent points. Explicitly, given a jump discontinuity at p such 
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that 



lim 



lim 



(5) 



we augment the relation {{p,k)}i by including the pairs with p associated to each of the 
integer scales between k' and k", i.e. the relation now includes {(po, k) \ k G {k', . . . , fc"}}. 
Because A; is a discrete variable, any change in k involves a jump discontinuity, but we are 
actually only adding points to fill in the gap when the difference is more than 1. We also 
include the points {(1, k) \ k G {0, . . . , fcj(l)}}, which fill in the gap to the p-axis, as well as 
the points {(0, k) | k G {m + 1, . . . }}, which extend the curve along the k axis, past k = m. 
The inverse relation, {{k,p)}i (Figure lb) is then transformed into a function by selecting 
the maximum value of p for each k, in the cases where there is more than one p for a given 
k. The resulting functions (Figure lc) are used as Ci(k) in Eq. (2). 

II. LIMITING CASES 

We evaluate the pairwise complexity profile for a number of limiting cases as examples. 

A. Ideal Gas (independent variables) 

First, we consider the case of m completely independent variables, with no mutual infor- 
mation. The coupling matrix A is the identity matrix: = 5ij, with each variable having 
complete coupling with itself (An = 1) and no coupling with any other variable (A%j^i = 0). 
Thus, for each variable, the only scale of action possible is one variable, which applies for 
any threshold up to and including complete coupling (p=l), i.e. 





Taking the inverse yields 





and summing over the m variables gives 




(8) 
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FIG. 1: Illustration of the inversion process of ki(p), as described in Section I: a. An 
example plot of a function ki(p). b. The inverse relation {(k,p)} for the example function, 
augmented with additional points to fill in gaps (squares), c. The resulting function 

C i (k)=p{k). 

In this case, there is no multiple counting because any potential "group" of variables has 
only one member, so Eq. (8) is in fact the complexity profile. If we explicitly apply the 
correction for multiple counting (Eq. 4), this is confirmed: 

1 m 1 

C(k = 1) = - [m - 0] + Ti [° - °1 = m ( 9 ) 

fc'=2 

and 

m 1 

c(k^i) = J2jj{°-°] = ( 10 ) 
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FIG. 2: Complexity profiles for two systems of three variables: a. Three independent 
variables, the "ideal gas" , as given by both the original and the pairwise formulations 
(Section II A). As discussed in Section V, this is also the pairwise complexity profile for 
three bits including one parity bit. b. Three coherent variables, the "crystal", as given by 
both the original and the pairwise formulations (Section II B). 



This result is intuitive. With complete independence, at the scale of one variable, the system 
information is equal to the number of variables in the system. With no redundancy, the 
system information is zero at any higher scale. The graph of the ideal gas case for m = 3 is 
shown in Figure 2a. 

B. Crystal (coupled variables) 

Next, we consider the case of m completely coupled variables. The coupling matrix A 
comprises only ones: = 1, Vi,j G Thus the scale of action available to each 

variable is the size of the entire system, even up to and including a threshold of complete 
coupling (p= 1), i.e. 



so 




(11) 




(12) 
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Thus, the inverse functions are 



, 1 k < m 

Q(k)={ ~ ,Vze[l,m] (13) 
k > m 

and summing these over the m variables yields 

. m k < m 

C(k) = { ~ (14) 
k > m 

Each variable is counting all of the other variables in its group of coordination for all scales. 
When we sum them, we are actually counting the same information for each of them. There 
is therefore no need to sum them, or, if we do, we divide the result by the size of the system. 
We explicitly use Eq. 4, to get this result: 

m-l 

C(l < k < m - 1) = ^ - [m - m] + — [m - 0] = 1 (15) 

k'=k 

and 

C(k = m) = — [m - 0] = 1 (16) 
m 

and 



C(k > m) = (17) 



so 



. 1 k < m 

C(k) = { ~ (18) 
k > m 

With all of the variables coordinated, there is only one bit of information at any scale. 
Because they are all coordinated, this information is represented at all scales up to the size 
of the system. The graph of the crystal case for m = 3 is shown in Figure 2b. 



III. PROPERTIES 



We prove that the pairwise complexity profile satisfies three properties: superposition 
of uncoupled subsystems, conservation of total degrees of freedom, and monotonicity. The 
first two properties have been shown to hold for the original formulation, and appear to be 
fundamental to the idea of the complexity profile [15, 16, 25, 26]. Monotonicity, on the other 
hand, does not hold in general for the original formulation [15, 21, 24]. 



9 



A. Superposition 



Given two uncoupled systems, Si and S 2 , with pairwise complexity profiles C l {k) and 
C 2 (k), respectively, the complexity profile of the combined system S — Si U 62 is given by 
C = C 1 + C 2 . The superposition principle is a generalization of the extensivity of entropy 
[15]. 

Proof: The coupling matrix of the combined system S, with elements indexed to corre- 
spond to those of Si and S2 in the natural way, is the block matrix 

Ai 
A 2 

where Ai and A 2 are the mi x rrii and m 2 x m 2 coupling matrices of Si and S 2 , respectively, 
as individual systems. Zeros are appropriately dimensioned null blocks. The scales of action 
of each variable in the combined system are its scales of action in its own subsystem, i.e. 

k}(p) < p; i G {1, • • .,mi} 
kf_ mi (p) < p; i e {mi + 1, . . . , m x + m 2 } 
because it has no coordination with any of the variables in the other subsystem. From the 
definition (Eq. 1), each of the fc?"(p)'s are bounded by < kjip) < mi and the fc 4 2 (p)'s are 
bounded by < kf(p) < m 2 and both are non- increasing. Thus, the inverse functions are 

C}{k) ie{l,...,mi} 
Cf-mi ( k ) i e ( m i + l,---,mi + m 2 } 

where we know that C}(k) = for k > mi and Cf(k) = for k > m 2 . Their sum is therefore 
decomposable into the two subsystems: 



Hp) 



(19) 



Ci(k) 



(20) 



mi+m,2 



mi 



7711+7712 



C(k)= C i (k) = J £cl(k)+ Cl mi (k) = C\k) + C 2 (k) (21) 

i=l i=l i=?7ii+l 

where C 1 (/c) = for k > mi and C 2 (k) = for k > m 2 and therefore C(k) = for 
k > max(mi,m 2 ). When we correct for multiple counting, the decomposition into the two 
subsystems is maintained: 

"11+7712 -< 



k'=k 

7711+7712 



C(k)= 77 C\k l ) + C 2 {k l )-[C\k l + l)+C 2 {k' + 1) 



k'=k 



mi+m,2 



(22) 



£ ^\c\k')-C\k' + l)}+ l\c 2 {k')-C 2 {k' + l) 



k'=k 
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and because C l (k) = for k > mi and C 2 (k) = for k > m 2 , this is 

mi m2 

c(ife) = ^ [c 1 ^') - c 1 ^ + 1)] + 2 p 2 ( fc/ ) - &{k ' + x )] = + ° 2{k ^ ( 23 ) 

which demonstrates superposition. 
B. Sum Rule 

The area under the complexity profile curve of a system depends only on the number of 
variables in the system, not on the dependencies between them. Therefore, given a number 
of variables, choosing a structure of dependencies involves a tradeoff between information 
at a large scale (redundancy) and information at a smaller scale (variability) [26] . 

Proof: The total area under the complexity profile curve is given by 



m m 



k k=l k'=k 

m 

= E>-r + 

k=l 

m 

= [C(k) - C(k + 1) 
k=i 



(24) 



= (7(1) - C(m + 1) 
= (7(1) =m 

where the last equality holds because we know that at a coupling threshold of one (the 
maximum threshold), there will be at least one variable meeting the threshold condition 
for each variable (namely, itself), so (7j(l) = 1. If there are other variables identical to 
the ith variable, it may be that fcj(l) > 1, but because of the way we defined the inverse 
function — in particular, filling in the gap to the p-axis — it is still the case that Cj(l) = 1. 
Therefore, from Eq. (3), 

m 

C(l) = X)l = m (25) 

i=l 

The notion that the organization of dependencies, which define the structure of a system, 
is equivalent to a prioritization of complexity at different scales has been particularly useful 
in applications of the complexity profile [15, 16, 25]. 
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C. Monotonicity 

The pairwise complexity profile is a monotonically non-increasing function of scale. This 
is not the case for the original complexity profile [15, 21, 24]. A complexity profile formal- 
ism that as a rule does not increase has the advantage of a certain intuitiveness. As one 
"coarse grains", i.e. increases the scale of observation, one expects a loss of information. On 
the other hand, the full complexity profile demonstrates that interesting phenomena such 
as "frustration" — e.g. the anticorellation of three binary variables with each other — are 
captured by oscillatory complexity profiles. Section V gives a simple example where this 
difference is manifest: the parity bit. We now prove that the pairwise complexity profile is 
monotonically non-increasing. 

Proof: The way we have defined complexity as a measure of information is such that if 
there is certain information present at a given scale, then that information is necessarily 
present at all smaller scales. From Eq. (1), any couplings that satisfy some threshold pi will 
also satisfy any lesser threshold p 2 < pi, which means that p 2 < P\ =>- fcj(p 2 ) < ki(pi), i.e. 
ki(p) is a non-increasing function. Its inverse Ci(k) therefore is non-increasing as well, as is 
the sum of those non- increasing functions, C(k). From Eq. (4), 



because C(k') > C(k' + 1). This proves that C{k) is non- increasing. 

IV. GENERALITY OF THE COMPLEXITY PROFILE 

The analysis thus far has been based on using the mutual information as the pairwise 
coupling. However, the properties derived are largely independent of the particular coupling 
function used, i.e. how the coupling strength is determined from the system state proba- 
bilities. This means that the properties derived in Section III arise from the conceptual 
and mathematical underpinnings of the complexity profile, rather than from the specific 
representation of the system. 

The constraints on the definition of the coupling function are minimal. In general, the 
coupling function must be normalized to vary between zero and one. For superposition to 
hold, "unrelated subsystems" must still be defined as having a coupling strength of zero 
for any pairs of components split between the two subsystems. For the sum rule to hold, 




1 



(26) 



12 



each variable must couple to itself with a strength of one (to be precise, it is sufficient that 
each variable couple to some variable with a strength of one). The mutual information has a 
logarithmic dependence on state probabilities, and its use in the pairwise formulation should 
lead to a good approximation of the original complexity profile. Another possible coupling 
strength function is the absolute value of correlation between variables, which has linear 
dependence on state probabilities. 

The generality of the complexity profile formulation is particularly meaningful for the 
sum rule property. In that context, it is helpful to consider the freedom of variation in the 
definition of C, rather than the coupling function. We can interpret C(k)/m as the average 
threshold of the coupling strength to imply a group of size k. The independence of the 
sum rule to different forms of this function means that the tradeoff between complexity at 
different scales is a valid feature of the complexity profile, regardless of how one identifies 
sets of elements as groups based on coupling strength thresholds. 

Physically, one might interpret this in the following way. An external force of a certain 
magnitude will destroy the weaker internal couplings between components of a system. 
Stronger couplings are less susceptible to being broken than weaker ones. As the strength 
of the force increases more couplings are progressively broken. However, different particular 
dependencies between the strength of the external force and the strength of the couplings 
broken are possible; it may for example be linear or logarithmic. What we have shown is that 
the sum rule is agnostic to the particular functional form of that dependence. Groupings 
that disappear at one scale due to increased sensitivity to the external force always reappear 
at another scale, maintaining the same sum over all scales. 

V. DIFFERENCES BETWEEN THE ORIGINAL AND THE PAIRWISE COM- 
PLEXITY PROFILES 

The pairwise complexity profile (using mutual information as coupling) and the original 
complexity profile differ for cases in which there are large-scale constraints on the states of 
the system that are not derivable from the pairwise interactions of the components of the 
system. Such cases have been termed "type 2 strong emergence" [24]. 

A simple example of such a system is that of the parity bit [24]. We have three binary 
variables, S = {x\,X2,xs}, in which one of the variables is a parity bit for the other two, 
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being if the sum of the other two is even, and 1 otherwise. Mathematically, X3 = x\ © £2, 
where © is addition modulo two, the exclusive or (XOR) gate. This system is symmetric — 
any one of the three bits serves as a parity bit for the other two. In this case, the possible 
states of the system are 
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The probabilities of the states of each pair 


of variables are therefore 
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where, for example, the probability of 00. is 


P(00.) = 


= P(001) + P(000) 


distributions of the states of the individual bits are 
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These distributions gives the complexity profile shown in Figure 3. 

However, note that the probability distributions of the individual bits and of the pairs of 
bits (T2 and T3) are identical to the distributions which would be derived from the case of 
three completely independent bits, giving the same pairwise correlation matrix, 

1 

A 10 (27) 
1 
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FIG. 3: Original complexity profile for three bits including one parity bit (Section V). 
Compare to pairwise complexity profile shown in Figure 2a. 

which gives the approximated complexity profile shown in Figure 2a. 

The pairwise complexity profile thus has limited applicability as an approximation to 
the original complexity profile in cases where large scale constraints dominate the fine-scale 
behavior of the system. However, when the dominant flow of information is the emergence 
of large scale structure from the fine-scale behavior, this approximation should be useful. 
We note that just as higher order correlations can result from pairwise dependencies (as 
in network dynamics), pairwise correlations can capture some higher-order dependencies, 
though not all. 

VI. COMPUTATION TIME 

One of the major motivations behind developing the pairwise formulation of the complex- 
ity profile is that the existing mathematical formulation is prohibitively expensive to compute 
for an arbitrary high-dimensional system. The original formulation requires a calculation 
that involves the mutual information of every subset of components, so the computation 
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time would grow combinatorially with the size of the system. No algorithm is known to 
perform the computation in time less than Q(N\). 

The pairwise complexity profile of a system of N variables can be computed by an algo- 
rithm whose most time-expensive step is ordering the iV couplings of each of the N variables. 
Because efficient algorithms can sort lists of N elements in 0(N log(iV)) time, the pairwise 
complexity profile scales as 0(N 2 log(iV)). 

VII. CONCLUSION 

We present a method of calculating a measure of complexity as a function of scale. The 
method is based on the concept of the complexity profile, which has proven to be useful 
for mathematically capturing the structure of a system of interdependent parts. The previ- 
ous mathematical formulation of the complexity profile, while powerful, is too demanding 
computationally for many real-world systems. By offering a set of related alternative formu- 
lations, we both provide an approximation tool and demonstrate that the principles of the 
complexity profile, including superposition and sum conservation, are more general than a 
particular formulation. 
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